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BRIEF REPORT 

Social Validity of the Social Skills Improvement 
System—Class wide Intervention Program (SSIS-CIP) 
in the Primary Grades 


Sarah Wollersheim Shervey Lia E. Sandilos 

University of South Dakota University of Virginia 

James C. DiPerna and Pui-Wa Lei 

The Pennsylvania State University 

The purpose of this study was to examine the social validity of the Social Skills 
Improvement System-Classwide Intervention Program (SSIS-CIP) for teachers in the 
primary grades. Participants included 45 first and second grade teachers who completed 
a 16-item social validity questionnaire during each year of the SSIS-CIP efficacy trial. 
Findings indicated that teachers generally perceived the SSIS-CIP as a socially valid 
and feasible intervention for primary grades; however, teachers’ ratings regarding ease 
of implementation and relevance and sequence demonstrated differences across grade 
levels in the second year of implementation. 


Impact and Implications 

The purpose of this study was to examine the social validity of the Social Skills 
Improvement System-Classwide Intervention Program (SSIS-CIP) for teachers in 
the primary grades. Participants included 45 first and second grade teachers who 
completed a 16-item questionnaire after implementing the SSIS-CIP in their class¬ 
room. Findings indicated that teachers perceived the SSIS-CIP as a socially valid 
and feasible intervention for primary grades; however, the program’s ease of 
implementation and relevance and sequence were rated differently across grade 
levels after teachers’ second year of implementation. 


Keywords: social emotional learning interventions, social validity 


During the past two decades, research has 
indicated that there is a strong link between 
students’ social-emotional skills and their aca¬ 
demic outcomes (Durlak, Weissberg, Dym- 
nicki, Taylor, & Schellinger, 2011). Unfortu¬ 
nately, many young children enter school 


without important social-emotional skills 
needed to successfully navigate their initial 
school experiences (Rimm-Kaufman, Pianta, & 
Cox, 2000). Moreover, fostering these skills in 
the early years of schooling is particularly es¬ 
sential, given that children with behavioral and 
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emotional issues may be less receptive to inter¬ 
vention by the end of second grade (Eron, 
1990). One promising way to promote social- 
emotional skills in the primary grades is through 
the use of classwide social-emotional learning 
(SEL) programs. 

A number of SEL programs have been devel¬ 
oped for use in the elementary grades, and many 
of these programs tend to have common fea¬ 
tures including explicit skill instruction and op¬ 
portunities to practice social and emotional 
skills (Collaborative for Academic & Social 
Emotional Learning, 2013). For these programs 
to produce positive outcomes, however, they 
must be implemented effectively (Berkel, Mau- 
ricio, Schoenfelder, & Sandler, 2011). Indeed, a 
growing body of research indicates that effec¬ 
tive implementation, or implementation fidelity, 
is critical for intervention outcomes (Durlak & 
DuPre, 2008; Kam, Greenberg, & Walls, 2003). 
Yet some large-scale SEL interventions have 
shown less-than-optimal levels of implementa¬ 
tion fidelity (Elias, Zins, Graczyk, & Weiss- 
berg, 2003), and it is possible that this lack of 
uptake is due to teachers’ perceptions of the 
social validity (i.e., usefulness, acceptability, & 
feasibility) of these interventions (Sanetti & 
Kratochwill, 2009). 

Forman et al. (2013) specifically examined 
implementation science within the field of 
school psychology and concluded that the com¬ 
mitment to intervention implementation was de¬ 
pendent on perceptions of acceptability, effi¬ 
cacy, and organizational resources. Given these 
findings, social validity appears to play a pivotal 
role in the effectiveness of school-based inter¬ 
ventions and should be considered as part of the 
evidence base for an intervention program. 
However, less is known about factors beyond 
the specific intervention components that influ¬ 
ence teachers’ ratings of social validity, such as 
previous experience implementing the interven¬ 
tion and developmental level. For instance, 
when teachers experience the efficacy of an 
intervention after implementing it with fidelity, 
this is likely to lead to continued implementa¬ 
tion (Forman et al., 2013). Additionally, as with 
any intervention, teachers may perceive some 
aspects of the program as working better with 
certain grade/age levels of students than others. 
Thus, it is important to understand teachers’ 
perceived social validity of an intervention, as 
well as factors that potentially contribute to 


these perceptions such as additional opportuni¬ 
ties to implement the intervention and develop¬ 
mental context. 

Social Validity of SEL Programs 

Social validity reflects the importance of hav¬ 
ing intervention methods and outcomes that are 
relevant and useful to the individuals enacting 
and/or experiencing the intervention (Kazdin, 
2005). Some researchers have studied social 
validity in terms of teachers’ overall perceptions 
of the ease of program implementation. For 
example, Buchanan, Gueldner, and Oanh 
(2009) examined acceptability of SEL programs 
by asking teachers how realistic it was for them 
to devote time to preparing and implementing 
SEL lessons. One third of participants indicated 
that one lesson a week was possible, but any 
more than that would become more difficult. 
Teachers also identified potential barriers to im¬ 
plementation such as preparation time, level of 
training, and difficulty teaching nonacademic 
activities in class. As another example. Wan- 
less, Patton, Rimm-Kaufman, and Deutsch 
(2013) conducted focus groups with teachers to 
identify supports and barriers to the implemen¬ 
tation of a specific SEL approach, Responsive 
Classroom. Teachers’ reported that certain fac¬ 
tors, such as being able to conduct the interven¬ 
tion at their own pace, would make the inter¬ 
vention more acceptable. 

In addition to the social validity of entire SEL 
interventions, some researchers also have exam¬ 
ined the acceptability, feasibility, and relevance 
of specific components of such interventions. 
For example, Schick and Cierpka (2005), asked 
teachers to complete a final social validity sur¬ 
vey after implementation of the Faustlos pro¬ 
gram, a German language version of the Second 
Step (Beland, 1988) program. The question¬ 
naire assessed program attributes such as clarity 
of the manual, suitability and user friendliness 
of the instruction booklet, correspondence of 
instructional scenarios to real-life situations, 
student opinions of the intervention, and per¬ 
ceived effectiveness of the intervention. Partic¬ 
ipating teachers rated all items as quite or very 
good, and 77% of teachers reported they would 
continue to implement the program. 

Social validity of specific intervention com¬ 
ponents also was examined by Webster- 
Stratton, Reid, and Stoolmiller (2008) who had 
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teachers complete satisfaction inventories dur¬ 
ing training and after implementation of the 
Incredible Years Teacher Classroom Manage¬ 
ment and Child Social and Emotional curricu¬ 
lum (Dinosaur School). The inventory assessed 
the usefulness and appropriateness of interven¬ 
tion components and materials (e.g., role plays, 
behavior plans), as well as their ability to inte¬ 
grate the program into the general curriculum. 
Results indicated that a large percentage of 
teachers were satisfied with the intervention 
components and felt they were able to integrate 
them into the general curriculum effectively. 
High ratings of social validity also paralleled a 
high degree of implementation fidelity, lending 
support for the link between social validity and 
implementation fidelity. 

Purpose of the Study 

The purpose of the current study was to ex¬ 
amine the social validity of a universal SEL 
program, the Social Skills Improvement System 
Classwide Intervention Program (SSIS-CIP; El¬ 
liott & Gresham, 2007) for primary teachers in 
first and second grade. Although the SSIS-CIP 
is commercially available for use in primary 
grades and has been shown to improve student 
social skills (DiPerna, Lei, Bellinger, & Cheng, 
2015; DiPerna, Lei, Cheng, Hart, & Bellinger, 
in press), no research has been published to date 
regarding the social validity of this universal 
program. In addition, few of the aforementioned 
SEL studies examined grade level differences in 
social validity, and none examined if social 
validity ratings changed over time as teachers 
gained experience with the program. Thus, a 
secondary goal was to examine if social validity 
ratings varied across grade level and years of 
program implementation. 

Method 

Participants 

Data for the current study were collected 
from all 45 first (n = 14) and second (n = 31) 
grade teachers who implemented the SSIS-CIP 
Lower Elementary Version (K-2; Elliott & 
Gresham, 2007) as part of a larger study fo¬ 
cused on program outcomes. Teachers were em¬ 
ployed in seven elementary schools in the mid- 
Atlantic region of the Unites States. Five of the 


elementary schools were from an urban district, 
and two were from a small rural school district. 
The majority of participating teachers were fe¬ 
male (84.4%), Caucasian (95.6%), and held a 
Master’s degree (68.9%). The demographics of 
the participating sample were similar to those of 
U.S. teachers (81.9% White, 76% female, and 
56% with at least a Master’s degree; National 
Center for Educational Statistics [NCES], 2012, 
2014). There were no significant differences 
between teachers in Grades 1 and 2 based on 
gender, race, or education level. 

Across the participating schools, 69.9% of 
students received free or reduced price lunch, 
and the racial/ethnic composition of the student 
population was approximately 65.8% Cauca¬ 
sian, 18.1% African American, 8.6% Hispanic, 
and 7.5% Other (i.e., Asian, Pacific Islander, 
Native American). Participating classrooms en¬ 
rolled a range of 20-25 students. The study was 
approved by the university’s Institutional Re¬ 
view Board. 

Measure 

At the end of each year, participating teachers 
completed a social validity questionnaire in 
which they were asked to rate their experience 
implementing the SSIS-CIP. Similar to the 
aforementioned approach of Schick and Cier- 
pka (2005), the social validity questionnaire 
was developed to assess instructional strategies, 
components, and materials specific to the SSIS- 
CIP. The questionnaire included 16 items with a 
response format rated as 1 (poor), 2 (fair), 3 
(acceptable), 4 (very good), and 5 ( excellent ). 
Items (see Table 1) inquired about teachers’ 
perceptions of the appropriateness and useful¬ 
ness of activities and materials (e.g., usefulness 
of student booklets as a teaching aid for les¬ 
sons), as well as the clarity, feasibility, and 
effectiveness of the intervention (e.g., From 
your perspective, how effective is the SSIS-CIP 
in promoting pro-social behavior?). The social 
validity questionnaire yields scores in six do¬ 
mains: ease of implementation, relevance and 
sequence, student behavior outcomes, booklets, 
role play, and videos. Across years, domains 
displayed high internal consistency (ot = .81— 
96; Table 1). The social validity domains also 
exhibited statistically significant moderate¬ 
positive correlations with observer ratings of 
implementation fidelity, r = ,35-.49, p < .05. 
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Table 1 

Item Content and Internal Consistency for Social Validity Questionnaire Domains 

Ease of implementation (a = .81—88) 

Amount of preparation required to implement the SSIS-CIP 
Clarity of the lesson plans provided in the SSIS-CIP teacher manual 
Feasibility of implementing the SSIS-CIP lessons in your classroom 
Relevance and sequence (a = .92-90) 

Relevance/importance of the 10 key skills taught within the SSIS-CIP Early Elementary Curriculum 
Sequence of the 10 units within the SSIS-CIP 
Student Behavior Outcomes (a = .89-95) 

From your perspective, how effective is the SSIS-CIP in promoting students’ prosocial behaviors (e.g., sharing work 
materials, helping a peer with their work, complimenting a peer)? 

From your perspective, how effective is SSIS-CIP in reducing students’ inappropriate classroom behaviors (acting 
out, off-task, etc.)? 

Booklets (a = .84-.85) 

Clarity of the student booklets 

Usefulness of the student booklets as a teaching aid for the lessons 

Appropriateness of language and activities in the student booklets for students in second grade 
Role play (a = .93-96) 

Extent to which the required role-plays correspond to real-life situations 
Students level of understanding of the role-play situations 
Effectiveness of the role-plays in promoting learning objectives 
Videos (a = .95-94) 

Extent to which the video examples correspond to real-life situations 
Students level of understanding of the situations presented in the videos 
Effectiveness of the videos in promoting learning objectives 

Note. SSIS-CIP = Social Skills Improvement System-Classwide Intervention Program. Cronbach’s a range is reported for 
Years 1 and 2. 


Procedure 

Data from the present study were collected as 
part of a multiyear efficacy trial of the SSIS- 
CIP. First and second grade teachers were re¬ 
cruited to participate in the trial after approval 
had been obtained by district superintendents 
and principals. Participating teachers were then 
randomized into SSIS-CIP implementation or 
business as usual conditions within schools. In¬ 
tervention teachers were formally trained in the 
implementation of the SSIS-CIP, and then they 
taught the curriculum to their students over a 
12-week period. 

The SSIS-CIP is comprised of 10 instruc¬ 
tional units that focus on 10 key classroom 
behaviors: following directions, listening to oth¬ 
ers, following classroom rules, asking for help, 
taking turns in conversations, cooperating with 
others, acting responsibly with others, showing 
kindness to others, ignoring peer distractions, 
and controlling temper in conflict situations. 
Each unit consists of three scripted lessons that 
focus on one of the key classroom behaviors. 
Each lesson requires approximately 20-25 min, 
and teachers use six instructional strategies (de¬ 


scribe, model, role-play, do, practice, monitor 
progress, generalize) to help students learn the 
specific target skill. In addition, each lesson has 
at least one brief (30s-90s) video vignette dem¬ 
onstrating the target skill within a classroom 
context. Other instructional resources, such as 
student instructional booklets and practice ac¬ 
tivities, are used to help facilitate the lesson. 

The social validity questionnaires were com¬ 
pleted in the spring of each academic year after 
all lessons had been administered. The ques¬ 
tionnaire was completed online, and teachers 
received $10 compensation for completion. So¬ 
cial validity data were analyzed across 2 years 
of intervention implementation. 

Data Analysis 

A two-way (2 X 2) mixed factorial analysis 
of variance (ANOVA) with repeated measures 
was run using SPSS version 24. In each analy¬ 
sis, grade level (Grade 1 or 2) was entered as the 
between-subjects variable and social validity 
rating by implementation year (Year 1 or 2) was 
entered as the within-subjects variable. A be- 
tween-within subjects interaction (i.e., Grade 
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Table 2 

Descriptive Statistics for Social Validity Domains by Grade Level and Year 
of Implementation 

Grade 1 Grade 2 

Year 1 Year 2 Year 1 Year 2 



M 

SD 

M 

SD 

M 

SD 

M 

SD 

Ease of implementation 

3.51 

.65 

3.30 

.68 

3.81 

.81 

4.31 

.60 

Relevance and sequence 

3.88 

.71 

3.50 

.91 

3.94 

.72 

4.41 

.52 

Student behavior 

3.04 

1.05 

3.30 

1.09 

3.50 

.86 

3.94 

.77 

Booklets 

3.40 

.71 

3.59 

.97 

3.73 

.89 

4.15 

.64 

Role play 

3.83 

.89 

3.83 

1.17 

4.08 

.80 

4.42 

.69 

Videos 

3.67 

1.28 

3.73 

1.15 

4.18 

.90 

4.24 

.78 


Level X Implementation Year) was also exam¬ 
ined. Correlational analyses indicated that there 
were no significant associations between the six 
social validity domains and the teachers’ edu¬ 
cation level, race, and the school in which they 
taught ip > .05). 

Results 

The distribution of the data approximated 
normality with acceptable ranges for skew 
(—1.11 to —.04) and kurtosis ( — .90 to .61). 
Descriptive statistics for social validity do¬ 
mains are reported in Table 2. Overall ratings 
of social validity fell in the acceptable to 
excellent ranges ranged (3.04 to 4.41) across 
domains, grade levels, and year of implemen¬ 
tation. The consistency of teachers’ item rat¬ 
ings across years fell in the moderate to high 
ranges (ICCs = .44-.82). Similarly, social 
validity domains exhibited moderate to high 
correlations (.53-.80) across years of imple¬ 
mentation. 

A two-way mixed factorial ANOVA was 
run for each of the six social validity do¬ 
mains. Levene’s test of homogeneity of vari¬ 
ance indicated equal variances for each social 
validity domain across grade levels and time 
points, with the exception of the relevance 
and sequence domain at Time 2, F(l, 23) = 
4.79, p = .04. Given this finding, two extreme 
outliers (2.5 SDs < mean) were identified and 
removed from the sample. Removal of these 
outliers resulted in a nonsignificant Levene’s 
test; however, results of the primary analyses 
yielded the same overall findings when the 
outliers were included in the sample. As such, 
the reported results reflect the complete sam¬ 


ple with outliers retained. For teachers’ rat¬ 
ings of the ease of implementation domain, 
there was a grade-level main effect, F( 1, 
22) = 7.81, p = .011, partial -q 2 = .26. This 
main effect was qualified by a statistically 
significant interaction between grade level 
and implementation year, F( 1, 22) = 12.00, 
p = .002, partial rf = .35. Specifically, first 
grade teachers’ ease of implementation rat¬ 
ings decreased slightly from the first to sec¬ 
ond year of implementation; whereas second 
grade teachers’ ratings increased slightly 
from the first to the second year (see Figure 
1). For the relevance and sequence items, 
there also was an interaction between grade 
level and implementation year, F( 1, 23) = 



Figure 1. Interaction between grade level and year on 
teachers’ ratings of ease of implementation. See the online 
article for the color version of this figure. 
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11.16, /? = .003, partial r| 2 = .33. Again, this 
interaction revealed that first grade teachers’ 
relevance and sequence domain ratings de¬ 
creased between the first and second year of 
implementation, while second grade teachers’ 
scores increased between the two years (see 
Figure 2). No main effects (p = .10-.96; 
partial rf = .00-. 12) or interaction effects 
(p = .18-.82; partial r| 2 = .00-.08) were 
found for the booklets, student behavior, role 
play, or videos domains. 

Discussion 

The purpose of this study was to examine the 
social validity of the SSIS-CIP Lower Elementary 
Version in first and second grade classrooms. 
Overall, teachers viewed the program positively. 
Social validity ratings in the various domains fell 
in the acceptable to very good range, with most 
falling above or approaching a rating of very good. 
These ratings are similar to ratings of social va¬ 
lidity reported for other social emotional learning 
programs. For example, teachers’ social validity 
ratings of the Strong Start Program (Gunter, Cal- 
darella, Korth, & Young, 2012) ranged from 
3.59-4.20 on a 5-point agreement scale (higher 
values indicated stronger agreement). Similarly, 
Webster-Stratton et al. (2008) reported mean 
social validity ratings ranging from 3.45-3.91 
on a scale from 1 ( unhelpful ) to 4 (very helpful). 



Figure 2. Interaction between grade level and year on 
teachers’ ratings of relevance and sequence. See the online 
article for the color version of this figure. 


While social validity ratings remained high 
across grade and amount of experience with the 
intervention, there was an interaction between 
year of implementation and grade level for two 
domains, ease of implementation and relevance 
and sequence. For both domains, social validity 
scores were slightly higher for second grade 
teachers during the second year of implementa¬ 
tion, but they were lower in the second year of 
implementation for the first grade teachers. This 
finding parallels the results of SSIS-CIP inter¬ 
vention efficacy trial in that the social skills 
outcomes (effect sizes) for first grade students 
were smaller than those of the second-grade 
students (DiPerna, Lei, Bellinger, & Cheng, 
2015; DiPerna, Lei, Cheng, Flart, & Bellinger, 
in press). 

Given the documented differences in student 
outcomes across grades and the link between 
perceived efficacy and acceptability of an inter¬ 
vention (e.g., Forman et al., 2013), it is perhaps 
not surprising that some differences exist across 
teachers’ ratings of social validity in multiple 
domains. Specifically, items in the ease of im¬ 
plementation domain assess clarity, feasibility 
of implementation, and amount of preparation 
required to implement, and items in the rele¬ 
vance and sequence domain assess the impor¬ 
tance of the skills targeted by the intervention 
and the appropriateness of their sequence. The 
observed differences over time across grade lev¬ 
els may reflect developmental differences 
across classrooms. While familiarity of the in¬ 
tervention appears to help with facilitation and 
understanding for second grade students, it may 
further highlight aspects of the SSIS-CIP that 
are perceived by teachers to be less useful for 
younger students. 

Implications for Practice 

There are several potential implications of 
the current study for practice. Regarding the 
SSIS-CIP, it is generally perceived by teachers 
as being useful, effective, and well organized 
for students in first and second grade. Nonethe¬ 
less, second grade teachers appear to view the 
program slightly more socially valid than their 
first grade counterparts. In addition, there are 
several practical considerations for schools im¬ 
plementing SEL programs. For example, it is 
important to consider grade-level variability in 
teachers’ perspectives regarding the social va- 
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lidity of universal SEL programs given the key 
role that teachers often play in implementing 
such programs. It is also important and poten¬ 
tially informative to assess the social validity of 
individual components of SEL programs to de¬ 
termine if some features of a program are per¬ 
ceived as less useful than others by key stake¬ 
holders. Such information can be used to 
potentially guide revision and future interven¬ 
tion development efforts. 

Limitations and Directions for 
Further Research 

There are several directions for future re¬ 
search based on the findings and limitations of 
the current study. First, although the current 
sample size was sufficient to detect large ef¬ 
fects, it provided insufficient power to detect 
small to moderate effects. Future studies of the 
SSIS-CIP should include larger and more di¬ 
verse samples of teachers and students to exam¬ 
ine its social validity across grade levels and 
years of implementation. Second, as the number 
of studies examining social validity have in¬ 
creased, it has been conceptualized and mea¬ 
sured in different ways. As such, another im¬ 
portant line of research is identifying the critical 
components of social validity that are associ¬ 
ated with implementation fidelity for SEL pro¬ 
grams. This, in turn, will facilitate the develop¬ 
ment of measures that are efficient, informative, 
and useful for program planning and evaluation. 
Finally, future research should identify strate¬ 
gies for increasing social validity and maximiz¬ 
ing teacher buy-in for SEL programs. Examin¬ 
ing teacher comments and ratings about existing 
SEL programs, as well as developing question¬ 
naires to help teachers identify what interven¬ 
tion components they think are most important 
in an SEL program, will help further inform 
future development efforts. 

Conclusion 

Ultimately, professionals tasked with choos¬ 
ing and implementing interventions are more 
likely to implement them with fidelity if they 
believe the intervention is acceptable, will 
work, and have the necessary resources for ef¬ 
fective implementation (Forman et al., 2013). 
Collecting information about how to enhance 
social validity of SEL programs is critical to 


their successful implementation, and thus the 
positive impact they can have on student out¬ 
comes. Results of the current study demonstrate 
that the SSIS-CIP appears to be a socially valid 
and feasible universal program for students in 
first and second grades. Though there were 
some observed differences in social validity rat¬ 
ings across grade level and years of implemen¬ 
tation, these differences require further investi¬ 
gation to determine if they are replicable and 
represent meaningful differences in social va¬ 
lidity across the primary grades. Given the 
growing interest in universal SEL programs for 
schools, social validity should continue to be 
explored for these programs to inform imple¬ 
mentation efforts, improve fidelity, and maxi¬ 
mize effectiveness. 
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